Strengths: This paper is quite amazing. Just adding a few linear layers causes otherwise standard, deterministic autoencoders to learn interesting generative factors of a similar or possibly greater quality to VAEs. It's rare to see a simple idea that works very well, with many possible extensions. I think this result will be of wide interest to the community. The theoretical observation that gradient descent dynamics in deep linear networks finds low rank solutions is well established, but has not been put to practical use.

This work got mixed reviews: R1 praised the potential impact of such a simple idea being shown to work remarkably well, but other reviewers had significant concerns about the empirical evaluation, which is especially important when the main contribution of the paper is to show that an idea is effective in practice. The reviewers were ultimately unable to reach a consensus about this paper, but all reviewers agreed that the core idea is promising, and R2, R3 and R4 raised their scores in light of the discussion and the author feedback. While the resulting scores still make this a difficult decision overall, I have chosen to recommend acceptance. The main point of discussion was whether the required changes to the manuscript require another review cycle or not. Indeed, the requested changes were quite broad: - demonstrate the effect of the initial variance of the linear layers - compare the model against modern autoencoder variants - compare against vanilla autoencoders with varying latent dimension - demonstrate the effect of the number of linear layers - avoid overclaiming, e.g. about the proposed model working well "with all types of optimizers" - etc.

An important component of autoencoder methods is the method by which the information capacity of the latent representation is minimized or limited. In this work, the rank of the covariance matrix of the codes is implicitly minimized by relying on the fact that gradient descent learning in multi-layer linear networks leads to minimum-rank solutions. By inserting a number of extra linear layers between the encoder and the decoder, the system spontaneously learns representations with a low effective dimension. The model, dubbed Implicit Rank-Minimizing Autoencoder (IRMAE), is simple, deterministic, and learns continuous latent space. We demonstrate the validity of the method on several image generation and representation learning tasks.

An important component of autoencoders is the method by which the information capacity of the latent representation is minimized or limited. In this work, the rank of the covariance matrix of the codes is implicitly minimized by relying on the fact that gradient descent learning in multi-layer linear networks leads to minimum-rank solutions. By inserting a number of extra linear layers between the encoder and the decoder, the system spontaneously learns representations with a low effective dimension. The model, dubbed Implicit Rank-Minimizing Autoencoder (IRMAE), is simple, deterministic, and learns compact latent spaces. We demonstrate the validity of the method on several image generation and representation learning tasks.